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Abstract. A classical theorem of R. Baer describes the nilpotent radical of a finite 
group G as the set of all Engel elements, i.e. elements y £ G such that for any x £ G the 
nth commutator [x, y, . . . , y] equals 1 for n big enough. We obtain a characterization of 
the solvable radical of a finite dimensional Lie algebra defined over a field of characteristic 
zero in similar terms. We suggest a conjectural description of the solvable radical of a 
finite group as the set of EngeUike elements and reduce this conjecture to the case of a 
finite simple group. 
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1. Introduction 



In the present paper we study certain Engel-like characterizations of the solvable rad- 
ical of finite dimensional Lie algebras and finite groups. Such a characterization for the 
nilpotent radical of a finite group is given by R. Baer (Baj . see also [Hj, [RoJ: 

Theorem 1.1. The nilpotent radical of a finite group G coincides with the collection of 
all Engel elements of G. 

This theorem of Baer generalizes Zorn's theorem jZoj which gives a characterization of 
finite nilpotent groups in terms of special two- variable Engel identities. 

Our goal is to establish an analog of Baer's theorem for the solvable radical of a finite 
group. As a first step, we consider a similar problem for finite dimensional Lie algebras. 



2 



BANDMAN, BOROVOI, GRUNEWALD, KUNYAVSKH, PLOTKIN 



Towards this end, we introduce some sequences of words which provide the solvability 
property. These sequences play the role of the Engel sequences that work in the nilpotent 
case. 

All groups in the paper are assumed to be finite, all Lie algebras are finite dimensional. 
Observe, however, that Baer's theorem is valid for many classes of infinite groups satisfying 
some finiteness conditions. For example, Baer himself proved it for noetherian groups, a 
similar result is valid for radical groups |Plolj . |Plo2j . linear groups |Plaj . and Pi-groups 
Plo3j (see also jPrj, fI5|). 

The paper is organized as follows. In Section 2 all main definitions and statements of 
the problems are given. Section 3 deals with the case of Lie algebras. In Section 4 we 
prove a reduction theorem for the case of groups. 

Acknowledgements. Bandman, Kunyavskh, and Plotkin were partially supported by the 
Ministry of Absorption (Israel), the Israeli Science Foundation founded by the Israeli 
Academy of Sciences - - Center of Excellence Program, and the Minerva Foundation 
through the Emmy Noether Research Institute of Mathematics. Borovoi, Kunyavskh and 
Plotkin were also supported by the RTN network HPRN-CT-2002-00287, and Kunyavskh 
and Plotkin by INTAS 00-566. 

We are very grateful to A. Merkurjev, B. Plotkin, A. Premet, M. Sapir, and E. Vinberg 
for useful discussions and correspondence. 

2. Definitions and main problems 
We follow the terminology introduced in |Plo3j . 

Let F 2 = F(x, y) be the free two generator group, W 2 = W(x, y) the free two generator 
Lie algebra. 

Definition 2.1. We say that a sequence u = u\,u 2 , Us, ... , u n , ... of elements from F 2 
is correct if the following conditions hold: 

(i) u n (a,l) = 1 and u n (l, g) = 1 for all sufficiently big n, every group G, and all 
elements a,g<EG; 

(ii) if a, g are elements of G such that u n (a,g) = 1, then for every m > n we have 
u m (a,g) = 1. 

Thus, if the identity u n (x,y) = 1 is satisfied in G then for every m > n the identity 
u m (x,y) = 1 also holds in G. 

A similar definition can be given for Lie algebras. 

Definition 2.2. For every correct sequence u define the class of groups (resp. Lie 
algebras) 6 = 0( u ) by the rule: a group (resp. Lie algebra) G belongs to if and only 
if there is n such that the identity u n (x,y) = 1 (resp. u n (x,y) = 0) holds in G. 

Definition 2.3. For every group (resp. Lie algebra) G denote by G( u ) the subset of G 
defined by the rule: g G G(u) if and only if for every a G G there exists n = n(a, g) such 
that u n (a,g) = 1 (resp. 0). Elements of G( u ) are viewed as Engel elements with respect 
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to the given correct sequence u . We call these elements l?-Engel-like or, for brevity, 
1?-Engel elements. 

Examples 2.4. 

(1) If u — e — ei, 62, 63, ... , where 

ei(x,y) = [x,y] = xyx~ x y~ x , . . .,e n (x,y) = [e n _i(x, y), y], 

then 0(1?) is the class of all Engel groups. In the case of finite groups the class 
0(e) coincides with the class of finite nilpotent groups |Zoj . 

Clearly, e -Engel elements of any group G are none other than usual Engel 
elements. If G is finite, the set G(e) coincides with the nilpotent radical of G 

(2) If u is determined by the following correct sequence of words: 

Ui = xy' 1 ,u 2 = ui(xy,yx) = [x,y], ...,u n = u n -i(xy,yx), 

then for finite groups the class 0(1?) coincides with the class of all finite nilpotent- 
by-two groups |BPj . 

(3) The correct sequence ~u, where 



u 1 (x,y)=x 2 y 1 x,...,u n (x,y) = [xu n _ 1 (x,y)x \ yu n „ 1 (x,y)y 



determines the class 0(1?) which coincides with the class of finite solvable groups 
|RCCKPPj . 

(4) The correct sequence s , where 

sx(x, y)=x,..., s n (x, y) = [s n _i(x, y)~ y , s n ^(x, y)\, . . . 

determines the class 0( s ) which also coincides with the class of finite solvable 
groups |B WWj . 

(5) The correct sequence w , where 

wi(x,y) = [x,y], . . .,w n (x,y) = [[w n _ u x], [w n _ u y]], . . . 

and [ , ] stands for the Lie bracket in a Lie algebra, determines the class 0(Tu) 
of finite dimensional solvable Lie algebras over an infinite field k, charfc 7^ 2, 3, 5 

It is easy to see that if g is a T?-Engel element in G then g is a T?-Engel element in 
every subgroup H containing g. If if is a normal subgroup in G and g is a ~u- Engel 
element in G, then g = gH is a 1?-Engel element in G = G/H. 

The following natural problem arises: 

Problem 2.5. Describe the class of sequences ~u such that 0(1?) is the class of finite 
solvable groups. 

Denote by F 2 (x, y) the n-th term of the derived series of the free group F 2 (x } y). 
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Proposition 2.6. Let ~u be a correct sequence. Then Q(u) coincides with the class of 
finite solvable groups if and only if 

(i) for every n there exists k = k(n) such that belongs to F 2 (x, y); 

(ii) there is no n such that u n (x,y) = 1 is an identity in one of the following groups: 
(1) G = PSL(2,F q ) where q > 4 is a prime power, (2) G = Sz(2 m ) ; m G N, m > 3 and 
odd, (3) G=PSL(3,F 3 ). 

Here PSL(m, ¥ q ) denotes the projective special linear group of degree m over¥ q . For 
q = 2 m we denote by Sz(g) the Suzuki group (the twisted form of 2 B2, see [HB, XI.3]^. 



Proof. The proof repeats the proof of Theorem 2.1 from BGGKPPJ and is based on the 



list of minimal simple non-solvable groups |Thj . □ 

Remark 2.7. Even for the case of the group G = PSL(2,F g ) the basis of identities is 
known only for small fields ¥ q and not known in general [CMSj , jSoj . The bases look 
highly complicated and do not provide any hint to check explicitly if a particular identity 
follows from the basis. 

Problem 2.8. Describe the class of sequences ~u such that 0( u) is the class of finite 
dimensional solvable Lie algebras. 

Denote by W^ix, y) the n-th term of the derived series of the free Lie algebra W-zix, y). 

Proposition 2.9. Let u be a correct sequence. Then Q(u) coincides with the class of 
finite dimensional Lie algebras over an infinite field k, char k ^ 2, 3, 5 if and only if 

(i) for every n there exists k = k(n) such that Uk belongs to W% (x, y); 

(ii) there is no n such that u n (x,y) = is an identity in the simple Lie algebra sLj. 

Proof. The proof immediately follows from the fact that if the characteristic of k differs 
from 2,3,5, then every simple Lie /c-algebra contains a subalgebra which has the algebra 
SI2 as a quotient (see |GKJNPj and references therein). □ 

Remark 2.10. The assumption charA; 7^ 3, 5 is technical and can probably be dropped. 

Remark 2.11. The situation with bases of identities for the minimal simple non-solvable 
Lie algebras is different from the group case. A basis of identities of the algebra SI2 over 
a field of characteristic zero is known and consists of two identities [Ra , [Bah] (see |MKj 



for the case where the ground field is finite, and KoJ and references therein for the case 
where the ground field is infinite of characteristic p). However, it is difficult to verify 
whether a particular u n is an identity in SI2 using this basis. 

Conjecture 2.12. There is a sequence ~u = u(x,y) such that for every finite group G 
the solvable radical of G coincides with G(u). 

Conjecture 2.13. There is a sequence u = u(x, y) such that for every finite dimensional 
Lie algebra L the solvable radical of L coincides with L(u). 
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Of course, it is highly desirable not only to prove the existence of such a u , but to 
exhibit an explicit sequence in each of the above cases (groups and Lie algebras). We give 
a partial positive answer to Conjecture 12.131 in Section El (see Theorems 13.71 and I3.12JI . 
and reduce Coni ecture 12 . 1 21 to the case of simple groups in Section 0] (see Theorem I4.9JI . 



3. Case of Lie algebras 



Let L be a finite dimensional Lie algebra over a field k. Denote by [, ] the Lie operation. 
For t G L the linear operator adt : L — > L is defined by (adt)x = [t, x\. 

By the solvable radical of L we mean the largest solvable ideal R of L (Bourbaki |Bouj 
and Jacobson jjj call R the radical of L). By the nilpotent radical of L we mean the 
largest nilpotent ideal N of L (Jacobson [J] calls N nil radical, and Bourbaki |Bouj calls 



it just the largest nilpotent ideal). 

Define the sequence e by e 1 (x,y) = [x,y] and, by induction, e n+ \(x, y) = [e n (x,y),y]. 
Then e n = (—&dy) n x. 

Definition 3.1. An element y G L is called an Engel element if it is "e^-Engel, i.e. for 
every x G L there exists n such that e n (x,y) = (i.e. (&dy) n x = 0). 

The following proposition is well known. 

Proposition 3.2. An element y G L is Engel if and only if sAy is nilpotent (i.e. there 
exists n such that (ady) n = 0). 

We need a lemma. 

Lemma 3.3. Let V be a vector space of dimension d over a field k, let x G V , and let 
A: V — > V be a linear map. If A m x = for some m, then A d x = 0. 

Proof of the lemma. By Fitting's lemma (see [Jj Sect. LL4]), V = Vq © Vi, where Vq 
and V\ are invariant subspaces of A, the restriction Aq of A to Vq is nilpotent, and the 
restriction A\ of A to V\ is invertible. Write x = (xq,Xi), where Xi G Vi (i — 0, 1). We 
have A™x\ = 0, hence x% = (because A± is invertible), i.e. x G Vq. Since Aq is nilpotent, 
we have Aq = where d = dimVo- We thus have A d °x = (because x G Vo). Since 
d > do, we conclude that A d x = 0. □ 

Proof of the proposition. If a.dy is nilpotent, then clearly y is Engel. Conversely, if y 
is Engel, then by Lemma 13.31 (ady) d x = for any x G L, where d = dimL. Hence 
(a.dy) d = and ady is nilpotent. □ 

Define the sequence by v\(x,y) = x and, by induction, v n+ i(x, y) = [v n (x,y), [x,y]]. 
Then v n+1 (x,y) = (-ad [x,y}) n x = e n (x, [x,y]). 

Theorem 3.4. Let L be a finite dimensional Lie algebra over a field k of characteristic 
zero. Then L is solvable if and only if for some n the identity v n (x,y) = holds in L. 
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Proof. If L is solvable, then V = [L,L] is nilpotent PI Cor. II.7.1]. Hence every pair 
z, t of elements of L' satisfies the identity (acU) m z = 0, where m = dimL'. On putting 

z = [x, [x,y]], t = [x,y], we get v m+2 (x,y) = 0. 

In the opposite direction, we mimic the proof of |GKNP[ Thm. 3.1]. Namely, we first 
reduce to the case when k is algebraically closed. This reduction immediately follows from 
the following easy lemma (cf. |GKNPj ). 

Lemma 3.5. Let k be an infinite field and let P(xi, . . . , x n ) be a polynomial in n variables 
over k. If P(xi, . . . , x n ) = for all xi, . . . , x n G k, then for any field extension K/k we 
have P(xi, . . . , x n ) = for all x±, . . . , x n G K. 

Proof. Since k is infinite, by jEJ Cor. IV. 1.7] P is the zero polynomial, and the lemma 
follows. □ 

Remark 3.6. A similar statement is valid for the elements of free associative and free 
Lie algebras, i.e. for non-commutative associative and Lie polynomials. 

Thus we now assume k algebraically closed and suppose that L satisfies v n (x,y) = 
and is not solvable. Denote by R the solvable radical of L. The algebra L/R is not zero 
and semisimple and also satisfies the identity v n (x,y) = 0. It contains a subalgebra S 
isomorphic to sl2 which also must satisfy the same identity. Being isomorphic to SI2, the 
Lie algebra 5* has a basis {e + , e_, h} such that [h, e + ] = 2e + , [h, e_] = — 2e_, [e+, e_] = h. 
Take x = e+, y = e_. We have [x, y] = h, hence V2(x, y) = [e + , h] = — 2e + , and for any n 
we get v n (x,y) = (— 2) n ~ 1 e + 7^ 0, contradiction. The theorem is proved. □ 

It turns out that the same sequence v n allows one to describe the solvable radical of L. 

Theorem 3.7. Let L be a finite dimensional Lie algebra over a field k of characteristic 
zero. Then its solvable radical R coincides with the set of all v -Engel elements of L. 

Proof. Let us first prove that every element y of R is v -Engel. Since char(A;) = 0, by jjj 
Thm. II. 7. 13], [L,R] is a nilpotent ideal. Hence every element t of [L,R] is Engel in L 
[jj Thm. II.3.3], i.e. for any z G L there exists n such that (&dt) n z = 0. Let x be an 
arbitrary element of L. The element [x, y] is Engel in L, hence on setting t = [x,y], z = x, 
we get v n (x, y) = for some n. 

Conversely, let us prove that every v -Engel element lies in the solvable radical. Let us 
first consider the case where k is algebraically closed. It is enough to prove that there are 
no nonzero T^-Engel elements in a simple fc-algebra L. 

Let L be a simple Lie algebra over k. Choose a Cartan subalgebra H C L and a Borel 
subalgebra B D H . Let 

L = H © L a 

ag<I> 

be the root decomposition. Here $ = $(L,H) is a reduced irreducible root system, and 
all the spaces L a are 1-dimensional. Let <3> + be the set of positive roots determined by B. 
We have 

L = L~ ®H®L + , where L + = L a , L~ = L_ a . 
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We write an element y G L in the form y = y + h + y + , where y G L , h G H, y + G L + . 
For every a G $ choose a nonzero element e a G L a . The set {e_ Q : a G $ + } is a basis in 
IT. Write ?T = E ae $+ c_ a (y)e_ a , where c_ a (y) G A;. 

Let yEL, y^O, y = y~ + h + y + . Since $ is irreducible, it has a unique maximal 
root; we denote it by 7. We may assume that H and B are chosen so that the coefficient 
c_ 7 (y) of y~ at e_ 7 is nonzero. 

Indeed, let Aut L denote the algebraic group of automorphisms of L, and let G be the 
connected component of AutL. It suffices to prove that there exists g G G(k) such that 
c^-y(gy) 7^ 0. Assume the contrary, i.e. that c_ 7 (gy) = for any g. Set V = {z G 
L I c^(z) = 0}, it is a subspace of L of codimension 1. Let W(y) denote the vector 
space generated by the elements of the form gy for g G G(k), it is a nonzero G-invariant 
subspace of L. Since by assumption W(y) C V, we conclude that H^(?/) 7^ -L. This leads 
to a contradiction, because the representation of G in L is irreducible for a simple Lie 
algebra L. 

So let y E L, y 7^ 0, and c := c_ 7 (y) 7^ 0. We shall find x such that v n (x,y) 7^ 
for any n. Take x = e 7 . Denote h 1 = [e 7 ,e_ 7 ], it is a nonzero element of H. We 
have [hry, e T ] = ae 7 with a ^ 0. Since 7 is the maximal root, we have [x,y] = ch^ + yf 
where yi G L + . Hence we have [x, [x,y]} = \e^^ch^ + y{\ = —cae 11 and, by induction, 
v n (x,y) = (— ca) n ~ 1 e 7 . Since a 7^ and c 7^ 0, we see that v n (x,y) 7^ for all n. Thus 
there are no nonzero "if-Engel elements in L. This proves our theorem in the case when 
k is algebraically closed. 



To reduce the general case to the case of the algebraically closed ground field, we shall 
show that if y is a ^-Engel element of a finite dimensional Lie algebra L, then one can 
choose n in the condition v n (x,y) = to be independent of x. Actually we shall show 
that one can choose n to be independent of x and y. 

Lemma 3.8. Assume that v n (x,y) = for some x,y G L and for some natural n. Then 
v d+i{x,y) = 0, where d = dimL. 

Proof. We apply Lemma 13.31 to the linear operator A = — ad [x, y]: z 1— > [z, [x,y]} in the 
linear space L. □ 

We can now complete the proof of Theorem 13.71 Let y G L be a ^-Engel element of L. 
This means that for any x G L there exists n such that v n (x,y) = 0. By Lemma [3. 81 we 
then get ^+1(2, y) — for any x G L. This is a polynomial identity in x. Since the field k 
is infinite, it follows from Lemma 1331 that t> rf+1 (x, y) = for any x G L, where L = L(&k. 
In other words, y is a V-Engel element of L. Since our theorem is already proved over 
an algebraically closed field, we see that y G R, where R is the solvable radical of L. But 
R = R® k k poUl Ch. I, §5, n°6], hence y G R. □ 

Remark 3.9. The sequence v is adjusted to the case of Lie algebras over a field of 
characteristic zero. Indeed, the key point in the proof of Theorems 13.41 and 13.71 was 
the fact that if L is solvable then [L, L] is nilpotent. This is no longer true in positive 
characteristic. Here is an explicit counter-example to the corresponding statements in 
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positive characteristic, based on the fact that [L, R] is no longer contained in the nilpotent 
radical of L. 

Example 3.10. We use an example given in [jj II. 7, pp. 52-53]. Let k be a field of char- 
acteristic p > 0, and L be a vector fc-space of dimension p + 2. Denote by {e, /, ex, . . . , e p } 
a basis of L and define a structure of Lie algebra by the following multiplication table: 



From formulas (j3.1)l it follows immediately that the subalgebra M = (e±, . . . , e p ) is an 
abelian ideal of L and the quotient S = L/M is a two-dimensional solvable algebra, hence 
L is solvable (see PJ loc. cit.}). We shall show that L does not satisfy any of identities 
v n (x, y) = 0. Indeed, take x = / + ex, y = e + e^. We have 

t ■= [ x ,y] = -e,vx(x,y) = x,v 2 (x,y) = [x,t]=e + e 2 ,v 3 (x,y) = e 3 , . . . , 

and, by induction, v p = e p , v p+ \ = ex, v mp+r = e r , .... Thus for all n we have 
v n (x,y) 7^ 0, and Theorem 13.41 fails in characteristic p. 

The same algebra L provides a counter-example in characteristic p to the statement of 
Theorem 13.71 because it is solvable but not all of its elements are ^T-Engel (for example, 
y as above is not). 

Let us now consider another sequence Tu which hopefully will be extendable to the 
case of positive characteristic. Define Wi(x,y) = [x,y] and, by induction, w n+ x(x, y) = 
[[w n (x,y),x], [w n (x,y),y]]. 

Theorem 3.11. |GKNPj Let L be a finite dimensional Lie algebra over a field k of 
characteristic different from 2,3,5. Then L is solvable if and only if for some n the 
identity w n (x,y) = holds in L. 

We hope that the same sequence w n allows one to describe the solvable radical of L. 

Theorem 3.12. Let L be a finite dimensional Lie algebra over an algebraically closed field 
k of characteristic zero. Then its solvable radical R coincides with the set of all vj-Engel 
elements of L. 

Proof. Let us first prove that every element y of R is w-Engel. Since R is an ideal 
of L, we have wx(x,y) G R for any x G L. From the definition of w n it follows that 
W2(x, y) G [R, R], and, by induction, w n (x, y) belongs to the (n — l)th term of the derived 
series of R. Since R is solvable, for some n we have w n (x, y) = 0. 

In the opposite direction, we have to prove that every iw-Engel element lies in the 
solvable radical. It is enough to prove that there are no nonzero -w-Engel elements in a 
simple k- algebra L. 

Let L be a simple Lie algebra over k. As in the proof of Theorem 13. 7[ we choose a 
Cartan subalgebra H C L and a Borel subalgebra B D H, and we choose e a G L a , 
e a 7^ 0, for all a G $ = R(L, H). We write an element y£iL&sy = y~ + h + y + , where 



(3.1) 



e,f) 




[e,ei] = e i+1 (l<i<p-l), [e,e p ] = ei, 
\f, e i\ = (* - !) e i (1<*<P), [ei,ej}=0 (l<i,j<p). 
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y G L , h G H, y + G L + . Write y = J2 a e$+ c -a{y)z-a- Let 7 denote the maximal 
root in $. 

Let y G L, y 7^ 0. As in the proof of Theorem \3.7\ we may and shall assume that 
c := c_ 7 (y) 7^ 0. We shall find x such that w n (x,y) 7^ for any n. Take x = e 7 . Denote 
/i 7 = [e 7 ,e_ 7 ], it is a nonzero element of H. We have [h y , e 7 ] = ae 7 with ^ 0, then 
[7i 7 , e_ 7 ] = — ae_ 7 . Since 7 is the maximal root, we have 

Wi(x, y) = [x, y] = ch^ + yf where y^ G L + . 

Hence we have 

[wi(x, y),x] = [c/i 7 + yf, e 7 ] = c[h 7 , e 7 ] = cae 7 
(we have [yf , e 7 ] = because 7 is the maximal root). Furthermore, 

c_ 7 ([wi(x,y),y]) = c_ 7 ([c/i 7 ,y]) = c_ 7 ([c/i 7 ,ce_ 7 ]) = -c 2 a, 

(here once again we use the assumption that 7 is the maximal root). As at the first step, 
we obtain 

w 2 (x,y) = [[wi(x,y),x], [w 1 (x,y),y\\ = -c 3 a 2 h 1 + y+, where y£ G L + . 

By induction we conclude that for all n we have 

w n (x,y) = (-c 2 a 2 ) n_1 c/i 7 + y+ where y+ G L + . 

Since c ^ and a ^ 0, we see that w n (x,y) 7^ for all n. Thus there are no nonzero 
■w-Engel elements in L. □ 

Remark 3.13. To extend Theorem 13. 121 to the case of any field k of characteristic zero, it 
is enough to show that any -w-Engel element y G L remains tu-Engel m.L = L®kk. To do 
that, it suffices to reverse the order of quantifiers in the definition of a -w-Engel element, 
i.e. to show that if y G L is a -w-Engel element, then there exists n (depending only on y) 
such that for all x G L we have w n (x,y) = 0. This would imply that y remains an Engel 
element in L. Indeed, with such a choice of n the relation w n (x,y) = is a polynomial 
identity in L (with respect to the variable x), and it remains true as an identity in L (cf. 
Lemma 13. 5|) . 

In other words, it is enough to show that the sequence A n (y) = {x G L : w n (x, y) = 0}, 
n = 1,2,..., of subsets of L stabilizes provided y is a w^-Engel element. In light of 
Theorem I3.12[ this is valid in the case when k is an algebraically closed field. We can also 
prove this fact for any uncountable field k, see Proposition 13. 14l below. 

Let k be a field. We fix an algebraically closed field Q containing k. Let S be a set 
of polynomials in the polynomial ring k[Xi, . . . ,X m ] in m variables. By a fc-closed set 
(or algebraic fc-set) in the affine space A m we mean the set B of common zeros in Q m of 
such a set of polynomials S (cf. [EJ Ch. IX, §§1,2]). In particular, we say that B = A m if 
B = Q m . By B(k) we denote the set of zeros of S in k m , i.e. B(k) = B n k m . 

Proposition 3.14. Let k be an uncountable field, and A m be an affine space over k. 
Consider a sequence of k-closed sets B n C A m , such that B\ C B% C . . . B n C . . . and 
Ui° Bi(k) = A m (k). Then there is n such that B no = A m . 
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Proof. We use induction on m. 

Step 1. If m = 1 then any fc-closed set is either finite or equal to A 1 . Thus, 

- either Bi are all finite, and Ui°-^i(^0 contains at most countable set of elements, 
which contradicts to the assumption B^k) = A m (k); 

— or there is n such that B no = A 1 . 

Step m. Assume that the claim is valid for all m! < m. Let x%, . . . , x m be the coordinates 
in A' m , and let H denote the hyperplane defined by the equation x\ = 0. 

Fix a point a = (0, a 2 , ■ ■ ■ , a m ) G H (k) and a line 

L a = {(t,a 2 , ...,a m ) \teQ}. 

Since U(B i r\L a )(k) = (UB^k)) D L a (k) = L a (k), there is n(a) such that B n ^nL a = L a 
(see Step 1). 

For a natural number I define a set 

D l = {ae H (k) : n(a) < 1} C if (A:). 

In other words, a & Di means that L a C B\. Let us show that D\ = V\{k) for some 
fc-closed subset Vi of H Q . 

Let 5; be defined in A m by a set of polynomial equations 

'Fi^X!, . . .,X m ) = J2 x lPi,lA X 2, ■ ■ -,30m) = 0; 
F 2> l(x 1 , X m ) = APi,2,l( X 2, x m) = 0; 

k Fsi,l( x li ■ ■ ■ j x m) = x lPi,si,l( x 2i ■ ■ ■ i x m) = 0. 

The condition L a C B\ implies that 

Fj,i{t, a 2 , . . . , a m ) = 

for any t G Q, i.e. Pi t jj(a 2 , . . . , a m ) = for all possible i, j. Thus 

Di = {a G H (k) : Pi,j,i(a) = 0}, 

and clearly D\ = Vi(k) for the fc-closed subset V\ of H defined by these equations. 

On the other hand, IJ^(^) = U A = H (k). Thus, by the induction hypothesis there 
exists Iq such that Vi = H , and, consequently, B io = A m . □ 

Corollary 3.15. Let L be a finite dimensional Lie algebra over an uncountable field k 
of characteristic zero. Then its solvable radical R coincides with the set of all Tu-Engel 
elements of L. 

Remark 3.16. The Proposition 13. 14l is not valid for countable fields. 

Here is an example. Let xi, . . . , x n , ... be the countable set of all the fc-points of the 
affine space A m over a countable field k for m > 1. Denote by L n the straight line 
containing x n and the origin, and set B n = |J™Lj. Then {J^° B n (k) = A m (k). 

Nevertheless, our conjecture is that Theorem 13.121 is valid for any field of characteristic 
zero. 
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Remark 3.17. Theorem 13 . 1 21 does not hold in positive characteristic because simple Lie 
algebras may then contain nonzero -w-Engel elements (see Example !3.18l below). However, 
we hope that the theorem remains true for "classical" Lie algebras (i.e. for those coming 
from algebraic groups). 

Example 3.18. Let L = W(l; 1) be the Witt algebra defined over a field k of char- 
acteristic p. Recall (see, for example, |SF| 4.2, p. 148]) that L is of dimension p with 
multiplication table defined on a basis {e_i, e$, ex, ... , e p _2} a s follows: 



If p > 2, the algebra L is simple jjSF, Thm. 2.4(1) on p. 149]. However, if p > 3, it contains 
nonzero wT-Engel elements. Indeed, let y = e p _ 2 , and let x = a_ie_i + • • • + a p _2e p -2 be 
an arbitrary element of L. From formulas (J3.2)) it follows that 



For p > 3 this implies [wx(x,y),y] = and hence w 2 (x,y) = 0. Thus y is a ty-Engel 
element, and the statement of Theorem 13 . 1 21 does not hold for L. 

In order to extend the characterization of the solvable radical obtained in Theorem 13 .121 
to a broader class of algebras, we suggest a little change in our original approach. 

Definition 3.19. We say that an element y G L is strictly Engel, if it is Engel and for any 
x G L the element [x,y] is Engel (see Definition 13. II for the notion of an Engel element). 

For finite dimensional Lie algebras the following counterpart of Baer's theorem holds. 

Theorem 3.20. Let L be a finite dimensional Lie algebra a field k, char(/c) = 0. The 
nilpotent radical N of L coincides with the set of all strictly Engel elements of L. 

Proof. By PI Thm. II. 3. 3] every element y of iV is an Engel element. Since N is an ideal, 
[x, y] G N for every x G L, y G N. Thus every element of N is strictly Engel. 

Let us show that there are no nonzero strictly Engel elements outside N. Let y be 
a strictly Engel element of L. Denote by R the solvable radical of L. First we prove 
that y G R . Indeed, if y ^ R, then by Theorem 13.71 there exists x G L such that 
(ad [x, y]) n x ^ for any n. Hence [x, y] is not Engel and therefore y is not strictly Engel. 

Thus y G R. The set of Engel elements of L which are contained in R coincides with 
N, see Poul Ch. I, §5, n°5, Cor. 7 of Thm. 1] (see also |SE1 Ch. 16, §4)]. Thus y G N. □ 

Remark 3.21. There is another approach allowing one to characterize the nilpotent 
radical in a slightly different manner. 

Let L be a finite dimensional Lie algebra, and L + be the corresponding vector space. 
Denote by (adL)* the subalgebra of the associative algebra End(L + ) generated by the 
linear operators of the form adx where x G L. Let y G L. Observe that the principal ideal 
(y) in the Lie algebra L generated by y consists of the elements u(y), where u G (adL)*. 

We say that an Engel element y is totally Engel if u(y) is an Engel element for every 
u G (adL)*. 



(3.2) 




Wx{x, y) = a_ie p _ 3 + a e p - 2 - 
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The next theorem follows from the definitions. 



Theorem 3.22. Let L be a finite dimensional Lie algebra over a field k. The nilpotent 
radical N of L coincides with the set of all totally Engel elements of L. 

Proof. An argument similar to that in the proof of Theorem 13.201 using J, Theorem 
II. 3. 3], shows that all elements of N are totally Engel. 

Conversely, if y is totally Engel, then the ideal (y) consists of Engel elements. By 
Engel's theorem, this ideal is nilpotent and therefore is contained in N. In particular 
y G N. □ 

Remark 3.23. Observations from the previous remark are also relevant to the case of 
the solvable radical and -w-Engel elements. We could define strictly (resp. totally) Tu- 
Engel elements in the same way as it was done for strictly (resp. totally) Engel elements. 
Moreover, to treat the case of positive characteristic, one may be led to an even more 
restrictive definition of totally -w-Engel element requiring that it remains tu- Engel after 
applying any (not necessary inner) derivation of L. In this setting one can hope to get a 
characterization of the solvable radical as the set of all totally ty-Engel elements. 

Remark 3.24. Using yet another approach, one can redefine the notion of strictly 
uT-Engel element using sequences of three (or more) variables. For example, define 
r n (x,y,z) = [z,[x,y], . . . ,[x,y)). Then an element y G L is strictly Engel if for every 
x, z there exists n = n(x, y, z) such that r n (x, y, z) = and e n (x, y) = 0. 



4. Group case. Reduction theorem 

Let G be a finite group. We have [x,y] = xyx^y^ 1 = xo-y^x^ 1 ) where a y G Aut G 
takes x to yxy~ x . 

Let u = ui(x,y), . . . , u n (x, y), . . . be a sequence in F 2 = F 2 (x, y). We want to define 
"iT-Engel-like automorphisms of G. 

Let G\ = G X Aut G be the semidirect product of G and Aut G. Recall that it can 
be viewed as the set of all pairs of the form (g,o~), where g G G, o G AutG, with 
multiplication (gi,o~i) . (#2, 02) = (5'i cr i(5 , 2), ^i^)- The natural embeddings G — > Gi and 
AutG — > G\, together with the above formula for the group law in Gi, imply that the 
equality aga^ 1 = o~(g) holds inside Gi, for any g G G and a G AutG (informally, an 
arbitrary automorphism of G becomes an inner automorphism inside a bigger group G\). 
In particular, this implies that aga^ 1 belongs to G. 

A u -Engel-like automorphism a of G should be defined in such a way that in the group 
Gi = G X Aut G it will be presented by a usual ^-Engel-like element. 

Let us take a correct sequence u(x,y) in F 2 (x,y). Given a group G, consider a ho- 
momorphism \i: F 2 (x,y) — > G\. Denote n(x) = g, fi(y) = a, where g G G, o G AutG, 
and define u n (g,a) = n(u n (x,y)) G Gi (informally, we "substitute" g instead of x and a 
instead of y). 
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Lemma 4.1. Suppose that a correct sequence u = u(x,y) in F 2 (x,y) satisfies the prop- 
erty: 

(Hi) For every x,y G F2 the word u n (x,y) belongs to the minimal subgroup containing 
the o~ y -orbit of the element x. 

Then for every group G X Aut G, every g G G, and every a G Aut G, the element 
u n (g,a) belongs to G. 

Proof. According to condition (iii), every element u n (x, y) can be represented as a product 
of elements yxy' 1 , y k x~ x y~ k . Then u n (g, a) is a product of elements a k g ±l o-~ k = c h (g ±l ) 
and thus belongs to G. □ 

Definition 4.2. We call a sequence u satisfying the hypotheses of Lemma [4.11 autocor- 
rect. 

We are now able to define ^u-Engel automorphisms: 

Definition 4.3. Let u be an autocorrect sequence in F 2 and let G be a group. We say 
that an automorphism a G Aut G is ~w-Engel if for any g G G there exists n such that 
u n (g,a) = 1. 

Example 4.4. For a G Aut (G) define [x,a] = xa{x^ 1 ). 

Consider the Engel sequence e defined by 

ei(x, cr) = [x, a},..., e n (x, a) = [e n _i(x, cr), a}. 

It satisfies the hypotheses of Lemma 14.11 We say that an automorphism a of G is Engel 
if for any x G G there exists n such that e n (x, a) = 1. 

Another example of a sequence satisfying the hypotheses of Lemma |4~T1 is w = w(x,a), 
where 

wi = [x,a], ...,w n = [[w n -i,x], [w n -i,a]\. 

Note that a = o y is a "u'-Engel automorphism if and only if y is a T?-Engel element. 
Thus if G has no non-trivial ^-Engel automorphisms, then G has no non-trivial ^-Engel 
elements. Let A be the cyclic subgroup of Aut (G) generated by cr. Denote G = G X A. 
If a is a Tf-Engel automorphism, then a is a u -Engel element in G. 

Let G be a finite semisimple group, i.e. R(G) = 1, and let ~u be an autocorrect 
sequence. Consider two classes of groups: 

• the class of semisimple groups with non-trivial ~z?-Engel elements; 

• the class of semisimple groups with non-trivial ~z?-Engel automorphisms. 

Our aim is to show that for some u these classes are empty. 

Any finite semisimple group G contains a unique maximal normal centreless completely 
reducible (CR) subgroup (by definition, CR means a direct product of finite non-abelian 
simple groups) which is called the CR-radical of G (see |Rol 3.3.16]). We call a product 
of the isomorphic factors in the decomposition of the CR-radical an isotypic component 
of G. 

Denote the CR-radical of G by V = V(G). This is a characteristic subgroup of C 
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Theorem 4.5. Let u be a correct sequence, and let G be a semisimple group of smallest 
order having non-trivial it-Engel elements. Then G has the following structure: 

(i) all non-trivial quotients of G are solvable; 

(ii) the CR-radical of G contains only one isotypic component. 

Proof, (i) Let H be an arbitrary normal subgroup in G. Denote G = G/H. Let R(G) = 
H\/ H be the solvable radical of G. Then G = G/R(G) is semisimple and of order strictly 
smaller than the order of G. Hence, for a "u'-Engel element g G G we have g — 1, and 
therefore g G Hi/H. Thus g G Hi. Since Hi is normal in G, it is semisimple. The order 
of H\ is smaller than the order of G and g G Hi, therefore g = 1. 

Thus G = Hi, and G/H = Hi/H is solvable. 

(ii) Take H = V(G), where V(G) is the CR-radical of G. Suppose V(G) =G x xG 2 , 
where Gi and G2 are isotypic components of V(G), i.e. products of isomorphic non- 
abelian simple groups. Then they are normal subgroups of G. By (i), G/Gi and G/G 2 
are solvable, hence so is G/(Gi D G 2 ). Since the intersection is 1, G is solvable. The 
contradiction shows that V(G) consists of one isotypic component. □ 

Observe that under hypotheses of Theorem 14.51 the CR-radical of G coincides with the 
intersection of all its proper normal subgroups. 

Let us now consider a group G which is minimal with respect to the second property 
(to possess non-trivial ^u*- Engel automorphisms). We need the following auxiliary result. 

Proposition 4.6. Let G be the group generated by Gq and go, and let Gq < G. Let G be 

a finite semisimple group, and Vq its CR-radical. If go acts trivially on Vq then g acts 
trivially on Gq. 

Proof. Denote R(G) = R, G = G/R. We have R n G = 1 and [R, G ] = 1, G = 
G R/R S G , V = V R/R S V . 

Consider the action of g on G and on G . We have goggo 1 = doggo 1 = <?o#<?o~ , g ^ ^o- 
Thus, the actions of go in Go and Go are isomorphic. Vo is the CR-radical of each of the 
groups Go and G. Suppose that go belongs to the centralizer of Vo. Then g — 1, and 
go G R(G), go and Go commute. Hence go acts trivially on Go- □ 

Proposition 14.61 implies that if a is an automorphism of a semisimple group G acting 
trivially on the CR-radical of G, then o is the identity automorphism of G. 

Theorem 4.7. Let u be an autocorrect sequence, and let G be a semisimple group of 
smallest order having a non-trivial ~u -Engel automorphism. Then G is a direct product 
of isomorphic non-abelian simple groups. 

Proof. Observe, first of all, that if a is a ^T-Engel automorphism of G and H is a o~- 
invariant subgroup of G, then the corresponding restricted automorphism o~i is a it- 
Engel automorphism of H. If now H is a normal subgroup of G invariant under a ~u- 
Engel automorphism a, then a, the automorphism of the group G/H induced by a, is a 
u- Engel automorphism of G/H. 



CHARACTERIZATION OF RADICAL 



15 



Let G be a minimal semisimple group which has a non-trivial u -Engel automorphism 
a. Let V be the CR-radical of G. If V < G then the restriction of a to V induces a 
u -Engel automorphism o\ of V which must be trivial. Then according to Proposition 
I4.6[ a is trivial on G. Contradiction. Thus V = G. 

Suppose V = G consists of several isotypic components V\, V2, . . . , V^. By the minimal- 
ity hypothesis on G, a acts trivially on each component, and a once again must be trivial. 
The contradiction shows that V consists of one isotypic component. □ 

Conjecture 4.8. There exists an autocorrect sequence ~u such that no group G of the 
form G = Y\ Gi, where all Gi 's are isomorphic simple non-abelian groups, has a non-trivial 
~u -Engel automorphism. 

Theorem 4.9. Conjecture \2.13l is equivalent to Conjecture \4 ■ 8[ 

Proof. First note that for any G and any correct sequence u all elements of R = R(G) 
are u- Engel. Hence the assertion of Conjecture 12.121 is equivalent to the following one: if 
G is semisimple, then it contains no non-trivial "u'-Engel elements. 

1. Suppose that u is a sequence such that for any group G we have R{G) = G(~Tt). 
Let G be any finite semisimple group, and let a be a non-trivial "u'-Engel automorphism 
of G. Let us show that a = 1. Let G = G \ (a), and consider G = G/ R{G). This group is 
semisimple, and a is a ^T-Engel element of the semisimple group G. Then by hypothesis 
a = 1, and a is trivial. Thus the statement of Conjecture 14.81 is fulfilled. 

2. Suppose Conjecture 14.81 is true, and take ~u as in its statement. We want to prove 
that there are no non-trivial 1?-Engel elements in any semisimple group. Assume the 
contrary, and let G be a semisimple group of smallest order containing a non-trivial 
"u^-Engel element g. Then a g is a non-trivial ^-Engel automorphism of G. According 
to Theorem 14.71 G = Y\Gi, where all GVs are isomorphic non-abelian simple groups. 
Conjecture 14.81 gives a g — 1. Hence g — 1. □ 

Remark 4.10. We believe that Conjecture 14.81 can be replaced with the following, a 

priori weaker 

Conjecture 4.11. There exists an autocorrect sequence ~u such that no finite simple 
non-abelian group G has a non-trivial ~u -Engel automorphism. 

Indeed, Theorem 3.3.20 of |Roj says that if G = Yli Gi, where all GVs are isomorphic 
simple non-abelian groups, then Aut G is isomorphic to the wreath product Aut Gi wr S n , 
where S n is the symmetric group of degree n. This gives some hope to deduce Conjecture 
14.81 from Conjecture 14.111 

Remark 4.12. Con i ect ures I4.1ll and l4.8l require to specify a sequence u . One of possible 
candidates is the sequence Tu from Example 14.41 
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